DISCRETE COMPONENTS IN RESTRICTION OF UNITARY 
REPRESENTATIONS OF RANK ONE SEMISIMPLE LIE GROUPS 

GENKAI ZHANG 

Abstract. We consider spherical principal series representations of the semisimple 
Lie group of rank one G = SO{n, 1; K), K = M, C, H. There is a family of unitarizable 
representations tt^ of G for jy in an interval on E+, the so-called complementary series, 
and subquotients or subrepresentations of G for jy being negative integers. We consider 
the restriction of {7T,y,G) under the subgroup H ~ SO{n — 1,1; K). We prove the 
appearing of discrete components. The corresponding results for the exceptional Lie 
group i^4(_20) ™d its subgroup Spin{8, 1) are also obtained. 



1. Introduction 

The study is of direct components in the restriction to a subgroup HcGofa represen- 
tation (tt, G) is one of major subjects in represenation theory. Among representations of a 
semisimple Lie groups G there are two somewhat opposite classes, the discrete series and 
the complementary series; the former appear in the decomposition of L'^{G) and can be 
treated algebraically, whereas the latter do not contribute to the decomposition and their 
study involves more analytic issues. The study of restriction of discrete series represen- 
tations has been studied intensively; see e.g. |fT9l USl and references therein. Motivated 
by some related questions of 13 Speh and Venkataramana ll23l studied the restriction 
of a complementary series representation of SO{n, 1) under the subgroup SO{n — 1,1). 
It is approved there, for relatively small parameter u (in our parametrization), the com- 
plementary series vTj, of SO{n — 1,1) appears discretely in the complementary series tTj, 
of SO{n, 1) with the same parameter u. They construct the imbedding of the comple- 
mentary series of SO{n —1,1) into of SO{n, 1) by using non-compact realizations of 
the represenations as spaces of distributions on Euclidean spaces and by extending distri- 
butions on M"~^ to M"^^. Similar results are also obtained for complementary series of 
differential forms. For n = 3 the same result is proved [|23l by using the compact picture 
and SU (2)-computations; see also ifTSl where a full decomposition is found. 

In the present paper we shall study the branching of complementary series of G for 
all rank one Lie groups G under a symmetric subgroup H. More precisely we prove the 
appearance of discrete component for G = SO{n, 1;K), H = SO{n — 1, 1; K), with 
F = M, C, EI being the fields of real, complex, quaternion numbers, or for G = -F4(_20) 
and H = Spin(8, 1) C G. We shall use the compact realization of the spherical principal 
series on the sphere S = K/M m¥^\ We prove that for appropriate small parameter 
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p the natural restriction map of functions on S in tx^ to the lower dimensional sphere S 
in F"~^ defines a bounded operator onto a complementary series vrjj, of H. The proof 
requires rather detailed study of the restriction to 5''' C S" of spherical harmonics on S. 

The representations Hi, for certain integers u have also unitarizable subquotients or sub- 
representations. Some of them are discrete series for SU{2, 1). We shall find irreducible 
components for the representations under the subgroup H. One easiest case is the sub- 
representation TT^ (or vr^^2 quotient) of the group SU{n, 1). The space ttq consists 
of holomorphic respectively antiholomorphic polynomials on C" modulo constant func- 
tions. It can also be treated by using the analytic continuation of scalar holomorphic 
discrete series at the reducible point [|7]|, and some general decomposition results have 
been obtained in [[T6l . These representations are also of special interests in automorphic 
representation theory ETI . 

We note that our result can be understood heuristically as certain boundedness property 
of the restriction map from certain Sobolev spaces on S to those on Indeed for small 
parameter u the space tTj, consists of distributions on S whose fractional differentiations 
are in L'^{S), i.e., they are functions with certain smooth conditions. It is thus expected 
that their restriction on the subsphere would make sense in proper Sobolev spaces. 
A precise formulation can be done and we hope to return to it in future. We remark 
also that the study of the norm estimates of the restriction of the spherical harmonics on 
lower dimensional spheres can be put into a general context as the study of growth of L^- 
norm of restriction on totaly geodesic submanifolds of eigenstates of Laplace-Beltrami 
operators on Riemannian manifold; see |[5l. Our results here give precise estimates of 
the L^-norm of the restriction. It would be interesting to see if there are similar type of 
results for general compact manifolds of positive curvature. 

I would like to thank B. Speh and T. N. Venkataramana for some correspondences and 
stimulating discussions during an AIM work "Branching problems in unitary representa- 
tions", MPIM, Bonn, July 201 1; they have obtained similar results for the case SO{n, 1) 
and SU{n, 1) earlier in an unpublished manuscript. I thank also T. Kobayashi, R. Howe, 
B. 0rsted and J. Vargas for several conversations. 

2. Preliminaries 

2. 1 . Classical rank one groups. Let F = M, C, EI be the real, complex and quaternionic 
numbers. Denote G := SOo{n,l;¥) = S'Oo(n, 1), ^[/(n, 1), S'p(n, 1) the connected 
component of group of F-linear transformations on F"+^ preserving the quadratic form 

kipH h|x„,p — |x„+ip, withF acting on the right. The group ii' := SOo{n), S{U{n)x 

U{1)), Sp{n) X Sp{l) is a maximal compact subgroup of G and G/K is a Riemannian 
symmetric space of rank one which can further be realized as the unit ball in F'\ Elements 
in G and q will be written as (n + 1) x (n + 1) block F-matrices 

a b 
c d 
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where a, b, c, d are of size n x n, n x 1, 1 x n, 1 x 1, respectively. 
Let = t + p be the corresponding Cartan decomposition. We fix 

- [l 0_ 

in p and let a = WRq C p. Then a is a maximal abelian subspace of p. The root space 
decomposition of q under Hq is 

= 0-1 + (a + m) +01 

with roots ±1, if F = M, and 

= 0-2 + 0-1 + (a + m) + 01 + 02 
with roots ±2, ±1, 0, if F = C, H. Here m C t is the zero root space in t. We denote 



the sum of the positive root spaces, in the respective cases. Thus m + a + n is a maximal 
parabolic subalgebra of 0. Let p be the half sum of positive roots. With some abuse of 
notation we write p{Ho) = p, and we have 



Denote M, A, N the corresponding subgroups with Lie algebras m, a, n. Then M = 
SOo{n — 1), SU (n — 1), Sp{n — 1) x 5^(1) and MAN is a maximal parabolic subgroup 



2.2. Decomposition of L'^{K/M). We identify p with F" and normalize the i^-invariant 
product on p so that Hq is a unit vector. The homogeneous space K/M is then the unit 
sphere S := S'''"^^ in p = F" with M being the isotropic subgroup of the base point 
Hq & p. We denote dx the normalized area measure on S and L^(S') the corresponding 
L^-space. For n = 1 the decomposition of L?'{K/M) is well-known and elementary, so 
we assume n > 1. Let be the space of spherical harmonics on S. For F = C let 
lyP'*^ be the spherical harmonics of degree p + g on C" and holomorphic of degree p and 
antiholomorphic of degree g. If F = H, then K = Sp{n) x 5^(1), and its representations 
are of the form ri x ri, which will be written as (ri, T2) and further identified with their 
highest weights. The root system of Sp{n) is of type C and let ai, ■ ■ ■ , a„_i, a„ be the 
simple roots with an the longest one. Denote Ai, ■ ■ ■ , An the corresponding fundamental 
weights with Ai the defining representation on C^". For 5*^9(1) = SU{2) the representa- 
tion on symmetric tensor power 0'^(C^) = will be written just as q for simplicity. 
Denote W^''' the representation (gAi + ^A2, q) of K = Sp{n) x Sp{l). 



n = 0i, tT = 0i+02 



(2.1) 




ofG. 
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Recall iniini 

e (WP,p>0 F = R 

(2.2) L^{S) = XI = I WP'^, P,q>0 F = C 

[WP''^,p>q>0,p-qe\en F = H 

Here and in the following we denote a general representation of K by r. The subspace 
(ly)^^ of A/-fixed vector is one dimensional 

where (pr is normalized by (pr{Ho) = 1. They depend only on the first variable Xi G H of 
a; = (zi, ■ ■ ■ , Xn), and will also be written as 0r(a;i)- They are given by 

(1) F = M, xi = cos^, 

:= cos^eF(-|, - tan^ 0; 

(2) F = C, xi = e^*cos^, 

= e^'^P-") 008^+" -g, n - 1, - tan^ 0; 

(3) ¥ = M, xi = cos ^e*^ = cos ^ (cos t + ?/ sin t) in quaternionic polar coordinates, y 
being purely imaginary and \y\ = 1, 

</>p,9(^) = - g.^^ cos^e F( , 2{n - 1), - tan^ 0- 

See [HH Theorem 3.1]. (Note that in the formula for ^l)p^q and ep,g in O p. 144-147] the 
term should be ^y^.) Here F(a, 6, c, x) is the Gauss hypergeometric function 2-F1, 

m=0 

and (a)m = njL^^('2 + i) is the Pochammer symbol. Note that all (/(-functions above are 
Jacobi polynomials |[24]| in t = — 1 in the interval (—1,1). 

We put the upper-index the dimension n as we shall also treat it as a variable. 

Li particular we have, by Schur's orthogonality relation, 

(2.3) ii0jr ^ 



dim(iy-)' 

dim(ty ) can be evaluated by the Weyl's dimension formula: Let {a} be the root system 
of I with {a > 0} the positive roots and the half sum of the positive roots, 

see e.g. JH. 

We shall also need a general integral formula: If f{x) = g{y)h{z), x = {y,z) are 
functions on M™ with separated variables y E M.'' and z E MJ^~'' with dy the Lebesgue 
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measure then we have 



(2.4) 



/i((l - \y\ )2z)dz dy 



m — k — l 



where dx and dz are the normalized area measures on the respective spheres and uJk~i = 
is the Lebesgue area of the sphere in M''' (we shall need k = 1,2,4 only); see 

e.g. [|20l 1.4.4 (1)] for the case of even m and k. Thus the square norm ||0r|P can also 
be proved by using the known integral formulas for Jacobi polynomials. However we 
shall use mostly the Weyl's dimension formula whenever possible as it is conceptionally 
clearer and as their asymptotic are well-understood. 

2.3. Exceptional group -F4(_2o). Let G be the connected Lie group of type F^_20) with 
Lie algebra g. This group has been well-studied Ifmi26ll . The maximal compact subgroup 
K is Spin{9) and the symmetric space G/K can be realized as the unit ball in with O 
being the Cayley division (octonian) algebra. Let g = p + the the Cartan decomposition. 
The space p will be identified with with t = spin{9) acting on via the Spin 
representation. We fix Hq E = p so that the positive roots of Hq in g are 2, 1. The 
corresponding multiplicities are then 7 and 8. The half sum of positive roots is p = IL 
Let ra be the zero root space of Hq in t, and m + a + n the maximal parabolic subalgebra. 

The algebra m C Hs spin(7). Let M = Spin{7) be the corresponding simply con- 
nected subgroup with Lie algebra m. Fix the i^-invariant inner product on p = with 
Hq being unit vector. The homogeneous space K/M is the unit sphere S = S^^ in 
= M}^. To describe the decomposition of L'^{S) under K we observe first that the 
space p = is decomposed under M as 

(2.5) p = O © O = {RHo © M^) © O 

with R'' being the defining representation of SO (7) and thus of M via the double covering 
M = Spin{7) — 7- 5*0(7), and O the Spin representation of M. The Dynkin diagram of 

Spin{9) is 

12 3 4 

O O Ct=«D 



with the simple roots ai,a2, 03, a^. Let Ai, A2, A3, A4 be the corresponding fundamental 
weights. Let W^''' be the representation of K with highest weight ^Ai + qX^. Then it 
follows inillll that 

(2.6) L\S) = ©p>,>0,p-,=Oevenl^^'^ 
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and each space W^''' has a unique Af-fixed vector (pp^g, (^W^''')^^ = ^4>p,q, such that 
(j)p^q{HQ) = 1. To describe y write elements in as x = {xo,Xi,X2) under the de- 
composition (12.51) . and write their (partial) polar coordinates as r = |a:|, a/xq + = 
r cos ^, xq = r cos ^ cos 7] with 0<^<|,0<?7<7r. Then 

(i)p,g{x) = 0p,g(a;o) 

= COS" V F{-\, ^; - tan^ r,) cos- ^ 4; - tan^ 0, 

for a; G S"; see ifTTI . 



3. Restriction of (50o(n, 1; F), tt^) to (50o(n - 1, 1; F) 

3.1. Principal series of G. For z/ G C let tt,^ be the induced representation of G from 
MAN consisting of measurable functions / on G such that 

(3.1) f{gme^^°n) = e'^^fig), e^"°mn e MAN 

and /|^ G L?'{K). (Our representation tTj, is Indf.jj^pf{e''~^'^) in the standard notation 
[fT4l . However the parameter has some advantage it is "stable" under branching; see 
Theorem 3.6 below.) In particular / in iii, are invariant under M, and Hi, is further re- 
alized on L'^{K/M) = L'^{S). We denote X^, the corresponding (g, -fT) -module with 
Xi, the algebraic sum of i^-irreducible subspaces in L?'{K/M). To simplify notation we 
shall denote ti^ also the corresponding unitary representation of G when [Xy^Tij^^Q) is 
unitarizable. 

The L^-norm in L'^(S) is not unitary for tt^ except when u = p + it for t G M, p being 
given by (12.11) . The unitarizable representations (X^, vr^, q) for real u are usually called 
complementary series. They have been found in lfT2ll . (See also (611 for related results 
for the real group SOo{n, 1).) The constant \u{t) below are rewritten in terms of the 
Pochammer symbol (a)m and further the Gamma functions. 

Theorem 3.1. There is a positive definite Q-invariant form on X^, given by 

\\w\\i = KiT)\\wr 

and its completion forms an unitary irreducible representation ofG, if 
(1)¥ = R,0 <u <n-l, 

(3.2) X,{p) 



(n — 1 — z/)p r{n — 1 — u + p) 



{u)p T{n-l-u)T{u + p)' 

(2) F = C, < z/ < 2n, 



(3.3) X,{p,q) 



in - Dp (n - f), _ T\^)T{n - | + p) r(n - f + g) 

r2(n-|)r(|+p) r(| + g) ■ 
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(3) F = e, 2 < < An, 



(2n- f)p_, {2n + l 

^ 1) v-q (77; P+g 



(3 4) ^ ^'2 

r(|-i)r(f) r(2n-f + g^)r(2n + i-f + g±g) 
~ r(2n-|)r(2n + i-|) r(| - 1 + £^)r(f + £±^) 

3.2. General criterion of boundedness. We fix n and let H = SOo{n — 1, 1; F) c G 

be the subgroup fixing the n-th coordinate x„ in F"+^ Denote L := K f] H, a maximal 
subgroup ofH. The subsphere S'^^''-^^-^ in F"-i of the sphere S = K/M c F" defined 
by the equation x„ = will be written as 5*^ which is homogeneous space of L, = 
L/L n M. To avoid confusion we denote by vrj^ the corresponding representations of H 
and the L-finite vectors , and the corresponding decomposition of L'^{S^) = L?'{L/ Lfl 
M) will be written as 



with a being specified accordingly. 

We shall need a general and elementary criterion for boundedness of intertwining op- 
erators. The sufficient part of the following Lemma 3.2 is used in [[22l implicitly, and we 
give here a proof for the sake of completeness. Let i^' be a temporarily a compact group 
and L c K a closed subgroup. Let (W, || ■ and (V, || ■ ||v) be two Hilbert spaces 
invariant under K and respectively L. Consider 



the irreducible decomposition of W and V under K and respectively L counting multi- 
plicities, all assumed being finite. Consider further the branching of W under L. Write 
cr C r if a representation a appears in r (counting multiplicities) with V'^''^ the corre- 
sponding irreducible component, and denote Pr^a the corresponding orthogonal projec- 
tion, i.e., 

e _ _ 

cr<Z.T 

Suppose _R is a densely defined L-invariant operator from iiT-finite elements in W to 
L-finite elements in V, and 

its components, i.e., R = X^r I^o-ct -^^-.o^ K-finite functions. We write ||i?||>v,v its 
norm whenever it is finite. 
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Lemma 3.2. The restriction operator R extends to a bounded operator from W to V if 
and only if there is a constant C such that for any a 

(3-5) ll-^r,a||vv.v ^ ^- 

Proof. Let w = J2t Wr EWhe an element with finite many nonzero components Wr- Its 
squared norm in W is 



|2 



c'^T II VV 



by our assumption. We compute the norm ||i?w||v. Writing w as w = J2t'^t 

EaCr ^r.^^r, WC haVC 

RW = R^^^P^^^Wr, 

and 

ll^^^llv = XI II 'Y ^r,aPT,aWr\\l < || ^r,a || >V,V || ^^.a^r || w)^- 

If the condition (13.51) is satisfied we find, by Cauchy-Schwarz inequality, that 

ii^^iiv<e(Eii^-.-iiw,v) (eii^- 

which is dominated by 

Cj^X] II ^r,aW^T II W = C'Y'Y\\Pr,aWr\\w = l^rHw = C'IkllvV' 

finishing the proof of sufficiency. Conversely suppose i? is a bounded operator. Then so 

is R*, and for a given v E V we have 

Cjl^il^ > ||i?*^||^ = WY^RI^^VII, = J2 WKrVWlv- 

But each i?* ^ is a scalar constant of an isometric operator by Schur's lemma, and we 
have 

l|-^cr,r^'llw — I|-^(T,tIIv,wII^IIv — ll-^r.o-llvV.vlK'llv- 



Substituting this into the above inequality we obtain (13.51) . □ 

3.3. Restriction of spherical harmonics. We specify the above considerations to the 
restriction R : ^ f{x', x„) ^ f{x'). The branching of = 

under L can be read off abstractly from known results. However we need to find all 
isotypic L-irreducible subspaces V^'" C with nonzero restriction, i.e. with the re- 
striction 

acting as an isomorphism. We shall drop the upper-index r in V'^''^ in the lemma below, 
as it is fixed in the summation. 
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Lemma 3.3. (1) F = R. The branching ofW^ under L = SO{n — 1) is multiplicity 
free. The restriction _q under L = SO{n — 1) is decomposed as 

® 

lx„=0 

0<q<p,p—qeven 

The corresponding unique q-isotypic component in is given by (as functions 
on S) 

V'^ = {hix')<p;-''{xn);heV^ 

(2) F = C. The branching ofW^'*^ under L = U{n — 1) is multiplicity free. The 
space WP''^\^ _p under L is decomposed as 

e 

pi<p,qi<q,'p-pi=q-<ii 

For each (pi, gi) the unique {pi, qi) -isotypic component in W'^''^ is given by 

(3) F = H. The space under L = Sp{n — I) x Sp{l) is decomposed as 

e 

]YP,q\ = ypi,qi_ 

'Xn=0 Z-^ 

p>pi,qi=g 

The corresponding {pi, qi) -isotypic component is given by 

Proof. Let F = M. The multiplicity free result in this case is well-known. The state- 
ment on the restriction is a result of Vilenkin ll27l (9), p. 495]. The proof there relies on 
explicit computations for the projection into spherical harmonics, which seem not easy 
to generalize to other cases. We give a slightly different proof which applies also to the 
other cases and which avoids some redundant computations. Denote L„ = X]J=i ^ the 

Laplacian on W\ Recall that the spherical polynomial / = r^Cm!^ (— ) is the unique 

n-2 

SO{n — 1) invariant polynomials on of degree m satisfying L„/ = 0, where Cm^ (t) 
is the Gegenbauer polynomial. Let x = G M", and put u := \x'\, v := x„. We 

have 

_ n-2 d 

OV^ OW^ U OU OV^ 

when acting functions depending only on \x'\ and a;„. Rephrasing in terms of u,v we 
have the unique solution of the form /(u, v) = {v? + v"^)^ C {-^j^^) , of the equation 



(3.6) i„; . f/ , + f/ . 0. 



n-2 



is when C = Cm ■ Now for fixed q < pwe search for an isotypic SO{n — 1) -component 
in of type consisting of homogeneous polynomials F{x) of degree p of the form 
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F{x) = h{x')f{u,v) = h{x')f{\x\',Xn), where h is a spherical harmonics of degree q 
on W'^^, i.e. Ln-ih = 0, and /(u, v) = (u^ + v^)'^C{^0^). The Laplace equation 
LrrF = {Ln-i + = bccomcs 



n-l 



{Ln-lh{x'))f{u, V) +2^ ^J ^g^^ '' -^/(^^ ^n) + h{x')Lnf{u, Xn) = 0, 

i=l ^ " 

with L„/ computed in (13.61) . But Ln-ih{x') = and X]j=i ^i^^^ = qh{x') by our 
assumption. Thus it reduces to 

1 dh 

(3.7) 2g-— /Kt;) + L„/(n,i;) = 0, 

i.e. 

d^f n-2 + 2qdf d^f 

This is precisely the equation (13.61) with n replaced by n + 2g and m replaced by p — g. 
Thus / is a constant multiple of rP~'^Cq ^ (y^) (which is a posterior polynomial in x). 
This proves the case for F = M. 

F = C. The multiplicity free result is also known; see e.g. lfT3l . The abstract decom- 
position of W^'^l follows easily by counting the degrees (pi, qi). The same argument 
above applies and we get that the polynomials of the type F{x) = h{x')rP~^''~^^~'^^C{^) 
as stated are spherical harmonics. Moreover they are of the right type (p, q) and thus they 
are in the space W^''^. 

¥ = M. The group 5^(1) acts on the space of polynomials on the right, h G 5^(1) : 
f{x) (-)■ f{xh), and it acts on the space W^''^ as the symmetric tensor ©"^(C^). So does it 
also on the space W^'"^ I . Thus any irreducible component must be of type I/pi'"?! with 

I Xn — U 

qi = q, again by (2.1). In particular p — pi = {p — q) — (pi — qi) is even since both p — q 
and pi — qi are even. This proves the decomposition. The rest of the proof is almost the 
same as above. (Note that the function o obtained from 0^ q in §2.2 by formally 
replacing n by n + ^, which is not necessarily an integer.) □ 



We compute now the norm of Rr,a- For positiva constants Cr^a and Dr^cr we write 
^ and ^ 



C D 

Cr,a ~ £'r,<x if both ^ and are dominated by positive constants independent of 



r, a. 



Proposition 3.4. With the notation as above we have the norm of Rr^a ■ W'^ C L^(S') — ?■ 
V C L\S^) is given by 

(1) F = M, j9 - g > even, 



\R: 



r(f) (2p + n-2)r(^^±£±2^)r(£^) p + i 



r(H^)r(^^±£±^) (p + g + l)i(p-g + l)r 

(2) ¥ = C, p > Pi > 0, Pi > qi > 0, p - q = Pi - qi, 

ll%,g),(pi,9i)f =p + q + n-l; 
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(3) ¥ = M, p ~ pi = 2k > and pi — qi > even, q = qi, 
T(2n — 2) 

\\RiP,,),in,J' = + ^^^^^ + ^^"^ -l)+pi-l){k + 2{n - 1) + p,) 

~ {k+l){k + pi + lf. 
In all other cases of{T, a) we have Rr^a = 0. 

Proof. F = M. By the previous lemma and Schur lemma we see that Rp^g : is 
up to a constant a partial isometry, and V'^ — )■ V is up to a constant an isometry. Thus 

\\R ll^ = Ml 

for any 7^ / G V'^. Now we take / = h{x')(j)p^q''{xn), which has a form of variable 
separation, and we have, by (12.41 ) and the g-homogeneity of h{x'), that 

11/11^ = rfn-Za^ I ^1 - Mr-^^'^\€^-l'M? f \hiy')Wdx^, 

and 

(3.9) l|i?/ir = l0p^f(O)P / Hy'W. 

Consequently 

\\Rpj' = p,5i?wiJ 0;-f(o)P( / (1 - \xn\r-^^''\<p;^?{x^)\'dx^r\ 



Note that the integral / := Jj^ — {xnl"^)" '^^ ~^'^\(j)p-q'^{xn)\'^dxn is up to a constant 
the square norm in L^(S'"'^^'^~^) of the spherical polynomial (j)p1^q'^{xn) in dimension 
n + 2g, and can be evaluated by using (12.31) in terms of the dimension dimiy^~|^ of the 
representation of SO{n + 2q). The exact (a rather subtle) constant is computed in (12.41) . 

■p/ n+2q-l \Y(l\ 

M</ ' "'^ (x„)|dx.- ^^^^^ 

Thus using the dimension formula that 

dimW^ _ , ^ + J - l^ [n + 3-^\ _{n + 2j- 2)r(n + j - 2) 

we have /^^ is 



J ; V J-2 ; r(j + i)r(n-i), 

2r(^) (2p + n-2)r(n + p + g-2) 



r(!l±f^)r(i) r(r2 + 2g - l)r(p - g + 1) 
The evaluation 0p+f(O) in (13.91 ) is zero unless p — q = 2k is even, in which case it is 



-1) 



k 
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But {-k)k = {-l)^k\, (-tti)fc = {-l)Hl)k = (-1)'^^^, we find that the evalua- 
tion, disregarding the sign (—1)'' and the constant r(i), is 



Using the product formula r(2x) = r(i)~^2^^'"^r(x)r(x + i) we obtain then the for- 
mula for II i?p^g II ^ as stated. The rest follows from the Stirling formula that 

r(?T, + a) 
V[n -\- b) 

The case F = C is done by similar computations. In the case F = EI we have 



,n+ 



Pi 



,p „2 i^p^pim'U\h{x')\'dx' 

\rC( 



2 



with Ul^p-l /if being 



2T(n) 



T(n - l)uj3 



1 Js'' 



. Pi 

by the integral formula above for separated variables. The norm of 0"^^" can not be 
computed using the dimension formula for pi odd as it can not be interpreted as spherical 
polynomials on a symmetric space. However we may use by known integral formulas If24l 
for Jacobi polynomials P^^^'^^t) on the interval [—1,1]. (More generally one may use the 
theory of Heckman-Opdam [|8| for Jacobi polynomials with general root multiplicities.) 
Indeed the function 0^ q in §2.2 for any real n > 1 can be written as 

r(fc + l)r(2n-2) (,„_3,) , 
<Pk,o{^)- Y{k + 2n-2) ^^'"^l 

where |x| is the norm of a quaternionic number x eM. The norm to be computed is 

/ i0:^,o(^)r(i - = c.3 f 1010(^)1^(1 - M^y^^'^-'^-'x^dx 

Jx£m,\x\<i Jo 
and which is further ll24l 

r2(2n - 2)r(A; + l)r(A; + 2) 



^^r{k + 2n- 2)T{k + 2n - l){2k + 2n - 1) ' 
The rest is done by a routine computation. □ 

Note that when F = M and n = 3 our result coincides with that in [l22l Lemma 2.4]. 
For F = C, and W'''^ = W^'^^ the space the holomorphic polynomials of degree p, the 
norm of R can be found directly by computing of the integral jx^pdx on the sphere S 
in C". 
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3.4. Discrete components of complementary series. Before stating our first main result 
we note the following elementary 

Lemma 3.5. Suppose 0<a<l,/3>0, a + /3>l and 7 > 1. Then 

00 00 

V- — T^<C^—, V- ^<C- -— , Vg>0 

The second estimate is straightforward. The first sum is dominated by the integral 

1 I""" 1 1 

dx = -, — — — — r / -, TTT^dx = -, — — -TT-rC 



Iq + g + 1)^ (g + 7o + 1)^ {q + 

since the integral x"{x+i)p '^^ = C < 00 is convergent by our assumption. 
Observe also that 

intertwines the action of nl of i). Thus the boundedness of R implies that (vr^, f)) is a 
discrete component whenever both are unitarizable. In accordance with the notation || ■ || 



in Theorem 3.1 we denote the norm of an operator T : — > and further 



= ||T||^,^. We have then 

IIP ||2 _ -^(^)\ | p ||2 

A(r) 

and the criterion (13.51) becomes 

(3.10) j:\\Rr,.rx{rr<^ 

Theorem 3.6. The restriction ofijiy^ G) on H contains (vr^, H) as a discrete component 
in the following cases 

(1) F = M, n > 3, < z/ < 

(2) F = C, n > 3, < z/ < n - 2; 

(3) F = H, n > 2, 2 < z/ < 2n - 1. 

Proof. ¥ = R. First note that '-^<n-2<n-l thus both {n^, G) and (vr,,, H) are 
well-defined unitary representations. We use now Lemma 3.2. The constants A(p), A''(g) 
and the series (13.101) in questions are 

Up) -{P+ At(g) ~ (g + 

P + l 1 



p>5,p— (jreven 



Writing p = g + 2j we see the sum is dominated by 

f> g + 2j + l 1 ^^f^ 1 



(2g + 2j + 1)5 (2j + l)Hg + 2j + l)"-i-2- - ^ jl (g + j)-i-2-| 



and further by (g + 1) ^ ^''^ in view of Lemma 3.5, namely by 
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¥ = C \u has the asymptotics 

For fixed type (pi, qi) of L the series J^rDa is dominated up to a constant 

by 

p+q+2 
P-Pi=q-qi>o ' ' I y 



E 



p— pi=g— gi>0 \j / \j y \j / 



as sum of two, say / + //. Now 

oo 



fc=0 

and 



by Lemma 3.5. The same holds for //. 

F = H. Writing p = pi + 2k,k>Q, we have 

and 

ll%,,),(pi,,)f ~(^ + l)(Pl + ^ + l)'• 
The sum (|3.10l) is bounded by 

{k + l){k+p, + lf 



< 



k+l 



E 

fc=0 



(pi - g + A; + + g + A; + l)2n-i-i^ 



A; + 1 

< 



i y 



{pi + q + 1)2"-!-^^ /--J _ g + A; + 1)2"+!-!' 

1 °° 1 



(pi + g + l)2n~i-'^ {j)^-qj^k + If n-u-i xKpi, qi) ' 
finishing the proof. □ 

Remark 3.7. For F = M and n = 3 the full decomposition of the complementary series 
Tiy of 5*00(3, 1) under 5*00(2, 1) is done in ifTSl . If (in terms of our parametrization) 
I < z/ < 1 it is a sum of two direct integrals of spherical principal series, and if < z/ < | 
there is one extra discrete component, the complementary series. The appearance of 
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complementary series tt^ of SOo{n — 1, 1) in the complementary series tt^ of SOo{n, 1) 
is done in ll23l using the non-compact realization on M"~^ It might be possible to find 
a full decomposition for general G = SOo{n, 1; F) using the techniques in [[TSl . The 
most interesting part might still be the the discrete spectrum in view of its stability under 
restriction and induction in the Ramanujan duals [l4l[3]|. 

3.5. The quotients (>V, vr,,) at negative integes u and their discrete components. The 

representation tt^ are reducible llT2l for v satisfying certain integral conditions, and there 
exist unitarizable subrepresentations (or quotients). More precisely we have the following 
result [|25l[I3, retaining the notation of as the Schur proportional constants; here we 
have rewritten them in similar formulation as in Theorem 3.1. 

Theorem 3.8. There is a unitarizable irreducible quotient {Wy^ Hy), whose completion 
forms a unitary irreducible representation ofG in the following cases 

(1) ¥ = R, n > 3, u = -k, k > 0, = X^/M^, 

k 

{n-l-u + k + l)p-k^i ^ T{n - 1 - u + p) 

(2) F = C, n > 2, 1/ = -2k, k>0,Wu = XJM^, 

P<k,q>0 q<k,p>0 

_ , , (n-f + A; + l)p_fc_i(n-f + + 
K{p,q) = - 



r(n-|+p)r(n-f + g) 



r(f +p)r(| + g) 

and for k = with three quotients {Wq, vr^), (Wo, hq), 

oo oo 
p=0 5=0 



and 



V{n + p) ^ V{n + q) 

oo 

Wo = Xo/^(iyP'° + l^°'^) 

p=0 

, / , Tin + p -l)T{n + q-l) 

^"^■"^^ fwfw ■ 
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(3)¥ = M,n>l,u = iy = -2k, k > -1, = XJM^, 

oo 

p~q<2k+2 p=q=0 

(2n - I + + l)E^_fc„i(2n + 1 - I + + l)p+9_fc„i 



[^-l + k + l)E^„fc„i(| + A; + l)£+9_fc_i 



r(2n-| + 2^)r(2n + l-| + £±2) 



r(f -i + £^)r(| + 2±2) 

Note that the same u as above is also a reducible point for {Xy^\,, tt^, t)). The corre- 
sponding quotient representation for f) will be written as (Vfc, tt^, f)) 

Theorem 3.9. Let n > Afar ¥ = R, n > 3 for ¥ = C, and n > 2 for ¥ = H. The 
representation (Vfc,7rJ^, [)) fanJ corresponding completion as for H) appears as an 
irreducible discrete component in (Wfc, '/r^(fc), g) (respectively for G) restricted to f) (resp. 
H). 

Proof Let Q = Q,, be the quotient map Q : ^ X^/Ml := Vk at the reducible point u 
as above for the group H. The map QR : X^ X^ Vk is clearly (7r|^, f)) intertwining 
and induces a map 

QR:W'' = XJM, ^ X^M^ = V^. 
We prove the boundedness of QR by the method above. Notice that the asymptotics 
exponent of A(z/) has the same dependence for positive u, e.g. in the case F = R with 
V = —k, 

A.(p)~(p + l)"-l-2^ p>k + i, 

and n — 1 — 2i^ > 2. Thus the same proof carries over to all cases, and we omit the 
details. □ 

There is some slightly difference when n = 3 for F = R, as = 5*00(2, 1) has its 
maximal compact subgroup being the torus and there is a spliting of the restriction to 
holomorphic and antiholomorphic discrete series. Note that we have also excluded the 
case F = C, n = 2, namely SU{2, 1), as the restriction map above is zero on the quotient 
>V_2fc; actually >V_2fc is a discrete series and it branching under SU{1, 1) can possibly be 
studied using some general tools [[T9l[T5l . 

3.6. The representation ttq and ttq for SU{n, 1). The representation vr^ on the quotient 

oo oo 
p=0 p=0 

are unitarizable representation of g. ttq can be constructed also by using the analytic 
continuation of the weighted Bergman space |l7l Theorem 5.4], i.e. scalar holomorphic 
discrete series, on the unit ball G/K in C" with reproducing kernel (1 — {z,w))~^ at 
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the reducible point fi = 0; see e.g. [[TOll where a reproducing kernel and its expansion 
are found for the space. A full decomposition under SU — 1, 1) of the series and their 
quotient can be obtained easily. Indeed let n^'^ be the corresponding representation for H 
and TTp'^ the unitary representation of H realized as the space of holomorphic functions 
on the unit ball {z G C"~^, < 1} with reproducing kernel (1 — {z, w)y, with H acting 
as 

G H, f{z) ^ {cz + d)-^f{{az + b){cz + rf)"^). 

TT^j^ is a discrete series of H only when j > n. Define analogously nj''" in terms of 
conjugate holomorphic functions. The following result follows from easy consideration 
of expansion of holomorphic functions f{x) in the last variable Xn- 

Proposition 3.10. The representation (vrj, G) is decomposed under H as 

oo 

i=i 

4. Restriction of {F^_20),'n-^) to H = Spin{8, 1) 

4.1. The subgroup Spin{8, 1). Recall that Hq e p = O'^ has nonzero roots ±2, ±1 in 
Q. Denote g±2 and q±i the respective root spaces. Then the Lie algebras q± generate 
a subalgebra of g of rank one which is easily seen to be f) := spin(8, 1). The Cartan 
decomposition of f) is f) = spin{8) © O with spin{8) acting on O by the spin represen- 
tation. The simply connected subgroup of G with Lie algebra f) is then H = Spin(8, 1) 
whose maximal compact group is L = Spin{8); see e.g. It follows from the de- 
composition (|2.5I) that the stabilizer of Hq E O C f) in if is also M = Spin{7) and that 
L/M = Spin{8) / Spin{7) is the sphere 5''^. We have thus 

where is the space of spherical harmonics of degree p on S'^, defined by the condition 

X2 = in S'-'^^ = {x = (xi,X2) G O^; |x| = 1}. The decomposition of (5*-^^) = 
L'^{K/M) is given in (|Z6|) with [W^^'^Y^ = Ccpp^g. We consider now the restriction of 
W^^'^|^2-o °^ components W^''^. 

Lemma 4.1. The decomposition of W^''^ under Spin{8) is multiplicity free and W^''^\^^^^ ■ 
V^; in other words the only irreducible component in the decomposition with non-zero 
restriction to is the representation V^. Moreover the square norm of R : W^''^ — 
is given by 

„ ^,,2 + n?=o(P + g + 8 + 2j)(p - g + 2 + 2j)(g + 4 + 2j)(g + 1 + 2j) 

W-tiW — Cy ; — ; 

" (g + 3)(g + l)5 

{p + l){p + q + lf{p -q + lY 



a b 

c d 
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where C is a numerical constant independent ofp and q. 

Remark 4.2. The representation W^''^ of K = Spin{9) is of highest weight + qXi 

with Ai, A2, A3, A4 the fundamental weights dual to the simple roots ai, 02, a^, a^. In the 
standard notation they are ai = ei — 62, 02 = 62 — 63, 03 = 63 — 64, = 64 and 
2^Ai + gA4 = |ei + |e2 + §63 + |e4 = ^{p, q, q, q). The four simple roots for spin{8) 
are (5i = ei — 62, ^2 = 62 — 63, ^3 = 63 — 64, ^4 = 63 + 64. The above branching rule 
above can be formulated as 

|yi(p,g,g,g)| ^ -^f (1,1,1,1) 

1x2=0 ' 

with ^/Ki'i'i'i) being the space of spherical harmonics of degree q on S*^, which is also 
of highest weight qei as SO (8) representation. This discrepancy of highest weights is 
explained by the triality in Spin(8). The Dynkin diagram of Spin{8) is 



4 




There is a symmetry of 5*3 (as outer automorphisms) acting on the three simple roots 
Si, S3, 64. The highest weight |(1, 1, 1, 1) = + 262 + ^3 + 254), whereas qei = 
|(25i + 2^2 + ^3 + ^4) and the permutation (134) exchanges the two weights. Also 
the multiplicity one property of W'^''^ under M = Spin{7) factors through Spin{8) and 
we have (WP'^y^l, _^ = _J*^ = (V)^'^ = 0^. Note that M = Spin{7) 

in Spin{8) C Spin{9) is not the obvious copy of Spin{7) in Spin{9) defined by the 
standard inclusion C C M^; in the space W^''' the former copy M = Spin{7) has 
only one-dimensional fixed vectors, whereas the latter copy Spin{7) has arbitrarily large 
multiplicities by the construction of Gelfand-Zetlin basis ll27l . 

Proof. The first statement is well-known. Any irreducible representation of Spin(8) in 
lyP''^!^ _Q is a constituent in L'^{S'^) and contains thus a unique M = S'j>m( 7) -invariant 

element. But {W^'^l = {W^'")^^ = C0P'«| and (jy^'^l is 

0p,<?(cos?7,O) = cos'' r]F{--, -; -tan^?]) 

which is precisely the A/-invariant spherical harmonics 0^(cos?7) on S*^, Section 2.2. 
Thus is nonzero and is just V. In particular the element 0p ^ is in the isotypic 

component V C W^''^ of V. The squared norm of R on W^''^ is 
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Both norms can be evaluated by the dimension formula. Following the notation in the 
above remark we have W^''^ has highest weight |ei + |e2 + fes + |e4 with the positive 
roots being {cj ± Cj, Cj, 1 < i < j < 4}, and the dimension of W^''^ is then 

2 

diml^^''^ = Clip + 7) f[{p + g + 8 + 2j)(p - g + 2 + 2j)(g + 4 + 2j){q + 1 + 2j) 

j=0 

~ (p + + g + lf{p -q + lf{q + if 
whereas the dimension of is 

dimV^ = C2{q + 3)(g + 1)5 ~ (g + 1)' 
for some constants Ci, C2 independent of p and g. This completes the proof. □ 

4.2. Discrete components. Define the principal series representation vr,^ of G as in (3.1), 
realized on L'^(K/M) = LF'^S^^). We recall results in [fTTI on the spherical complemen- 
tary series of G. 



Theorem 4.3. Let 6 < 1/ < 16. There is a positive definite (g, n^) -invariant form on the 

(g, K)-module ^ W^^'^ defined by 



\w 



\v = \/K{pa)\\w\\, \y{p,q) 



;^f-3)p^ (|)£±2 



We study now the branching of the complementary series under H = Spin{8, 1). 
Denote ttJ^ the principal series representation of 500(8, 1), thus also for H, as defined 
for in (13.11) . Note that there is a discrepancy between the normalization of Hq here and 
there; the roots of Hq in f) is ±2 here instead of ±1 as in Section 2.1. In particular, 
the restriction map R : /(xi,X2) H- Rf{xi) = /(xi,0), (xi,X2) G O^, defines an 
f) -intertwining operator 

i?:(X,,7r,,g)->(Xt,7rt,()). 

Theorem 4.4. Lef 6 < z/ < 7. T/ie restriction of {iXy^ G) on H contains (vrt , H) as a 
discrete component. 

Proof. The and in this case are 

K{p, q)-{p-q + iV'^^iP + g + 1)'^-^ A^(g) ~ (g + 1)'-'' 
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with p — q = 2k > even. The sum to be treated is 

q + k + 1 



^ (fc + 1)8-^^(2^ + g + 1)8-'' 
5^ {k + iy-''{2k + q + iy-'' 



k=0 

oo 

< 

k=0 



1 ^ 1 

- (g + 1)7-!^ + 1)8-^ 



(5+1)'-" AtM' 

2 

completing the proof. □ 

Remark 4.5. The complementary series in is parametrized in [j3i Example C] as — 5 < 
A < 5, i.e. the standard parametrization [fT4ll . Our u is their p + A = ll + A. Itis stated 
there that the point A = 3, i.e. = 8 is in the automorphic dual Gam of G. Note that this 
point falls outside the range 6 < < 7 in our theorem. One can deduce from the Burger- 
Li-Sarnak conjecture on the Ramanujan dual -ffRaman for H = SO{n, 1) and our theorem 
above on some nonexistence of certain intervals in the set -f/^Raman- In view of [3, Theorem 
1] it would be also interesting to study the induction of automorphic representations of H 
toG. 

The representation tTj, has also unitarizable subquotients at integral u: for u = 6 — 2k, 
A; > 0, the quotient 

p—q<k 

is unitarizable; see ifTTIl . However in this case the restriction composed with quotient map 
is zero. Presumable there is no discrete component under H and it would be interesting 
to pursue this further. 

The main results in the present paper prove the existence of one single discrete compo- 
nent under Hofa complementary series of G. It may happen that there are more discrete 
components. We shall study them in a forthcoming paper. Applying the stablity result jSl 
Theorem 1 (ii)] to those cases where nl appear discretely in tTj, we conclude finally if Tr,y 
is an automorphic representation of G so is vrj^. 
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